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THE EVOLUTION OF EVALUATION: TITLE I TO CHAPTER I 



BACKGROUND ! 

In the Johnson White House of 1965, Congress enacted the Elementary and 
Secondary Education Act (ESEA). The largest allocation of ESEA funds was 
targeted to the then Title I of the Act, and indeed historically Title I now 
Chapter I has emerged as the largest federally supported elementary and 
secondary program in the nation's history. 

Appropriation ^ for the program have grown from $959 million in Fiscal Year 
1966 to st3.48 billion in Fiscal Year 1984. The legislation whether 
initially Title I of the Elementary and Secondary Education Act (ESEA) and 
now Chapter I of the Education Consolidation and Improvement Act of 1981 
(ECIA) has as its primary purpose to provide financial assistance to local 
educational agencies in order to meet the needs of children who are 
educationally deprived and who live in high impact areas of poverty. This 
piece of social legislation will come before the Congress again in 1987 for 
reauthorization hearings. Twenty years plus experience with com|>ensatory 
programs have yielded a proliferation of experiences, field studies, 
questions — both answered and tmanswered and a number of significant 
observations as cited by Mary Kennedy (unpublished paper, 1984). 

1. The characteristics of the poverty population have changed since 
the program began 20 years ago, and these changes have raised 
questions about the nature and extent of students* need for 
compensatory ec'ucation* 

2. The prograui*s effects on students have been measured repeatedly 
over the life of the program, and a variety of methods have been 
used to assess its impact. The weight of this evidence now 
appears to indicate that, on the whole, the program has a marginal 
effect on student achievement* This evidence raises questions 
about the appropriateness of the current structure of compensatory 
education programs* 

3. There has been a considerable growth in knowledge about effective 
educational practices for Disadvantaged children and about 
ef f ecti /e educational practices in general . That knowledge has 
raised questions about the extent to which practices known to be 
effective are being implemented in compensatory education 
programs, and about the extent to which such practices are 
feasible with the program's legal and administrative structure* 

ISSUE ! 

Independent of (1) congressional reauthor''*':ation hearings, (2) questions on 
the hill, (3) pressures from constituent groups^ lobbyists, educators at the 
federal, state and local levels, and (4) ar ^untability demands from the 
General Accounting Office and the like, for twenty plus years the issue has 
been quite succinct — does the program work? Obviously, this question is 
then parsed out to smaller sub-questions i*e. 
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Who is served by the program? 
What are the services provided? 
Hov well are the services delivered? 
What do the services cost? 

What are the effects of services on recipients? 
What are the costs and benefits of alternatives? 

The information gathered through the evaluative process is obtained to 
facilitate making judgments or decisions about some aspects of the program. 
The audiences for the information depend partly on which questions are 
answered and may include policy makers, managers, oversight groups and the 
like. This description is of course deceptively simple. Matte7s become 
complex once the decision to evaluate is made. With the Title I/Chapter I 
Program the following issues have a significant impact on the design and 
conduct of evaluation studies and on the utility of the evaluation outcomes: 

• Title I is not a national program in the true sense of the word. 
Rather it is a funding source for local educational agencies. 
LEAs as the eligible recipients design and implement their own 

programs*' consistent with federal and state guidelines. 

• Achievement gains may be diff: :ult to attribute to Title I since 
high risk children receive many overlays of federal and state 
compensatory programming. 

• On a national level, when one speaks to the effectiveness of 
Chapter I programs, one is dealing with aggregated evaluative 
information from well over 14,000 different programs 
operationalized at the local level. Individual programmatic 
differences are "washed out" in the aggregate and quality control 
problems with data and data edits as well as sampling errors tend 
to be aggravated and enhanced at the natiox^al levels. 

• Evaluation requirements have also shifted, changed and evolved 
from the inception of the ESEA legislation in 1965 to the present. 

The purpose of this paper is: (1) to trace the evaluation requirements of 
the Title I/Chapter I Program since 1966 to the present; (2) to discuss the 
implications of the evaluations; and, (3) to report on and to discuss the 
evolution of evaitiation during the 20 year period in question. 

It is a premise of this paper that evaluations generally moved in four 
phases from a nadir of low quality, utility and precision in the mid to late 
sixties (Phase I) to a transition point during the early to mid seventies 
(Phase II) to a point of high quality, utility and technical rigor in the 
late seventies with advent of the implementation of the Title I Evaluation 
and Reporting System (TIERS) (Phase III) to a relaxation of standards and 
qualitv given the revised requirements of ECIA (Phase IV). The picture 
could be viewed much like a sine curve, with current evalixation quality, 
performance and efforts slipping to the down cycle of the curve. Negative 
as this may be, the sine curve is cyclical and one can only hope that after 
reaching a negative amplitude, that the sine curve again will be on the 
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upswing. Perhaps the amplitude bands will tighten given the past behavior 
of the evaluation sys' em as a whole. Pictorially, this could be represented 
as follows: 
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Figure 1 

Subjective as this pictograph may be in terms of definitions and shape of 
the curve, there is informal agreement amongst evaluators and program 
managers at local, state and federal levels that the quality of Chapter I 
evaluations is on the decline. 

PHASE I: THE INCEPTION OF THE ESEA TITLE I PROGRAM 
EVALUATION FROM 1966-1974 



Title I of the ESEA of 1965 (P.L. 89-10, as 
P.L. 95-561) was enacted to provide: 



amended by P.L. 93-380 and 



.financial assistance. ..to local educational agencies serving 
areas with concentrations of children from low-income families to 
expand and improve their educational programs by various 
means... which contribute particularly to meeting the special 
educational needs of educationally deprived children. . (ESEA, 
Title I, Section 101, "Declaration of Policy") 

Evaluation of program impact was always a requirement of the federal law and 
specific evaluation set asides were provided for in the legislation. 
Delineated below are the evaluation requirements prescribed in the 1966 
legislation and a description of the evaluation impact and issues associated 
with the implementation of these requirements. (See Table I) 
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PHASE II: TRANSITION YEARS 

EVALUATION FROM 197A-1978 



Given some of the problems cited above respective to the quality of the 
evaluation process for Title I, specifically the lack of comparability and 
validity of the data, Congrecs enacted Section 151 of ESEA Title I in the 
Education Amendments of 1974 (?•!• 93-380) • Section 151 requirements were 
essentially maintained in Section 183, when the Education Amendments of 1978 
(P.L. 95-561) were passed. Presented below are the evaluation requirements 
which were specified in Section 183 of P.L* 95-561 and a description of the 
evaluation impact* (See Table II) 

PHASE III: THE TIERS YEARS 

EVALUATION FROM 1978-1981 

The TIERS years marked the high point of technical quality a ' rigor with 
respect to the Title I program at the local, state and national levels. By 
the time the TIERS system was mandated in 1978, school districts across the 
nation had already begun phasing in the new models. During the 197C-1977 
school year, 20 states had implemented TIERS on a pilot basis. Ii^ 1977-78, 
nearly every state had identified local school districts who were field 
testing an identified evaluation model. With the advent of 1978-79, most 
states had begun to fully implement TIERS and to work closely with their 
Technical Assistance Centers (TACs) and with the U.S. Office of Education. 
Tne implementation of TIERS was not without its debates on technical rigor 
and programmatic issues. 

Although many state and local educational agencies far surpassed TIERS, in 
terms of both design and sophistication of their evaluation models, 
nonetheless, TIERS catalyzed positive movement for the field of evaluation. 
State agencies without clear mandates or guidelines for program evaluation 
used the TIERS system as a vehicle for evaluating not only Title I programs 
but other compensatory evaluation programs. In addition to Title I, other 
federally funded as well as state funded programs during the seventies also 
carried with them clear directives for program evaluation. These evaluation 
set-asides prompted advancements and movement in the overall technology of 
program evaluation. It is apparent that such advances would not have been 
possible without this tjrpe of external stimulus or indeed pressure. (See 
Table III) 

PHASE IV: POST TIERS 

EVALUATION FROM 1981 TO PRESENT 

The Education Consolidation and Improvement Act of 1981, was passed by 
Congress on July 31, 1981. All chapters of the act took effect beginning 
October 1, 1982 and were authorized through FY 1987. It was the Reagan 
administration's intention through the block grants to shift control over 
educational policy away from the federal government back to state and local 
agencies. The budget authority for the block grants in 1982 was reduced to 
80 percent of the total sum of the combined programs in 1981. 
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Tlie basic components of the Act consists of three chapters. Chapter 1, 
"Financial Assistance to Meet Special Educational Needs of Disadvantaged 
Children/* is a rewrite of ESEA Title I. The new language draws upon 
provisions in the old ESEA Title I law such as retaining the allocation 
formula and the emphasis on low-income children, but also provides seme 
simplification and relaxation of previous requirements* 

Traditionally, categorical programs have had a strong political lobby and 
vested interest group to provide their support bas . It is no secret that 
continued funding for these programs has been guaranteed not on the outcomes 
of the program or evaluation data but by legislation. Accountability is 
vested in the cloak of regulation, guidelines and procedure. The 
preservation of an intact Chapter 1 — "Financial Assistance to Meet Special 
Education Needs of Disadvantaged Children" is a manifestation of the power 
and concern of the special interest group's for disadvantaged studeits. 

For block grant programs, funds must be allocated and utilized ^ti accordance 
with law, statute and administrative regulation. Accountability for the 
block grants took three basic forms: 

1. descriptive reporting of clients served, fiscal accounting, 
services offered including program description and implementation 
process, 

2. impact (effectiveness) evaluation, and 

3. monitoring of programs by advisory groups. 

Over the years, educational evaluation, accountability and reporting efforts 
at national, state and local levels have matured as witnessed by TIERS the 
Migrant Student Record Transfer System (MSRTS) and many others. The 
aforementioned are offered as only illustrative examples without comment on 
technical rigor, adequacy or total acceptance by the evaluation community. 
The point being made is that evaluation, accountability and program 
monitoring were coming into their own as tools for enlightened 
decision-making and program planning end redirection. The sophistication in 
evaluation is due primarily to muturation of the field itself but in no 
small part is also due to the valuation demands placed on state and local 
program managers by the power of federal statute and regulation. Indeed, 
federal evaluation regulations have done much to catalyze lethargic state 
and local agencies into more structure, rigor and technical accuracy and 
adequacy with respect to evaluation planning, implementation and utilization. 

Current provisions for evaluation and reporting in the Education 
Consolidation and Improvement Act of 1981 are minimal but pose significant 
evaluation problems at the local, state and national levels. (See Table IV) 

SUMMARY ; 

The block grant legislation has the ongoing tension of insuring 
accountability while giving grantees operational flexibility. The Oninibus 
Budget Reconciliation Act and the policy of the current Administration have 
sharply curtailed Federal participation in block grant administration. The 
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Federal government has chosen to rely more heavily on accountability 
mechanisms that give a retrospective view or program accomplishments than on 
others that might involve it more directly in program decision-making. 
Depending on how these are implemented, and depending on the scope and 
nature of the States' voluntary efforts to establish and maintain common 
reporting systems, data collected across the Nation may not be comparable. 
This may affect the Federal Government's ability to ascertain progress 
toward its national objectives, should it choose to do so. (Government 
Accounting Office, 1982) 

The block grant enacted in 1981 places responsibility for program evaluation 
at the State level. If the history of the Federal agencies under the 
earlier block grants can be used to predict the problems that the States 
will encounter, one would expect the States to differ substantially in the 
manner and vigor with which they pursue program evaluation. Varieties of 
strength in current evaluation functions and perceptions about 
accountability may also make for differences among them. Funding problems 
associated with recent cutbacks in Federal aid may sharply curtail State 
evaluatfons despite the mandate for State evaluation activities. 
(Government Accounting Office, 1982) 

Tracking federally supported activities, recipients, and dollars is a major 
evaluation function. Whether Federal funds support activities that advance 
national objectives is historically of central interest to the Congress. 
Tracking weaknesses in the earlier block grant programs aroused 
congressional concern and led to the creation of management information 
systems and other such mechanisms. However, the changes in Federal and 
State responsibilities for evaluation under the new block grants may have 
opened a gap in the ability to assess nationally how well block grant 
programs achieve the national objectives that the legislation was designed 
to address. (Government Accounting Office, 1982) 

Evaluation systems must be applied uniformly across the States if comparable 
data are to be collected and analyzed. At present, the Federal agencies are 
not requiring that uniform data be collected on the 1981 block grants, 
although voluntary efforts are under way in some areas. The availability of 
an authoritative future source of national information about (1) the nature 
of program operations, (2) the levels and types of service available, and 

(3) the effect of programs on the problems they are intended to address 
remains in doubt. (Government Accounting Office, 1982) 

With the Chapter I program, the history of evaluation evolution is readily 
apparent. We are on the declining slope of the sine curve from an 
evaltiation perspective. A number of current efforts are currently under way 
which will shed more light on the outcomes of Chapter I programs. A large 
grant award was made to the National Institute of Education (now part of the 
U.S. Department of Education) to study the Chapter I program in all of its 
aspects. The planned activities of this effort include: (1) synthesizing 
recent data on state agency responses to ECIA, Chapter I; (2) a survey of 
school districts; (3) analysis of existing school district records; 

(4) collection of research syntheses; (5) nature of administrative 
activities; (6) review of the history of audit exceptions; (7) influence of 
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federal, state and local agencies on .their respective policies; (8) review 
of studies of state compensatory education programs; (9) district procedures 
for choosing schools and students; (10) study of local allocation of 
resources; (11) processes by which districts and schools design and deliver 
services; (12) survey of schools; and, (13) study of students' whole school 
day. This report will be ready in time for the congressional 
reauthoris:ation hearings for the Chapter I legislation. The General 
Accounting Office is also in the process of looking at the evolution of 
evaluatioii from Title I to Chapter I. It will be interesting to see the 
results of these investigations as they bear on the program effectiveness 
issue. Independent of the findings, the constituent lobby for serving the 
educational needs of disadvantaged children is strong and is a major force 
to contend with on the Hill in terms of action for this social program that 
has sustained itself for the past 20 years. As all evaluators know, 
decisions are often made with or without data in i socio-political 
environment. As an evaluator would that we could harken back to the rigor 
of TIERS. It remains to be seen what role evaluation played in the climate 
of reduced evaluation and reporting requirements at the national, state ad 
local levels. 
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TABLE I 
Evaluation from 1966 - 1974 



EVALUATION REQUIREMENTS 

An LEA could receive Title I funds 
only if: 

"...procedures are adopted for 
evaluating. . .the effectiveness of the 
programs assisted under this title in 
meeting the special education needs of 
educationally deprived children; 
.*.such evaluations will include... 
objective measurements of educational 
achievement in basic skills..." (Sec- 
tion 124(g), ESEA Title I) 

In turn, the SEAs must assure ED that: 

••Each State Educational agency shall 
make ... periodic reports. .^evaluating 
the effectiveness of programs assisted 
under this title and of particular 
programs assisted under it in 
improving the educational attain- ment 
of educationally deprived child- 
ren... •• (Section 172, ESEA Title I) 



IMPACT AND ISSUES 

Title I reports did not provide a 
consistent basis of information 
regarding the impact or effectiveness 
of the '^itle I program nationally. 
Differences in: 

(1) style and substance of the 
report; 

(2) evaluation methodology; 

(3) assessment techniques; 

(4) types of participation, and 

(5) achievement data. 

made the possibility of aggregating, 
synthesizing and utilizing the data 
at a national level near or nil. 
Additionally, it was also not clear 
if evaluations conducted at the local 
level would indeed be useful in terms 
of providing information on program 
strengths, impact or educational 
benefits to the students. (English 
et al., 1982 and Rossi, et al., 1977) 
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TABLE II 



Evaluation from 19 7 A - 1978 



EVALUATION REQUIREMENTS 

"••v(a) Independent Evaluations* - 
The [Secretary] shall provide for 
independent evaluations which 
describe and measure the impact of 
programs and projects assisted tinder 
this title. Such evaluations may be 
provided by contract or other 
arrangements, and shall be made by 
competent and independent persons* 

"•••(b) Evaluation Standards and 
Schedule. - The [Secretary] shall (1) 
develop and publish standards for 
evaltiation of program or project 
effectiveness in achieving the 
objectives of this title, and (2) 
develop. a schedule for conducting 
evaluations. • • designed to ensure 
that evaluations are conducted in 
representative samples of the local 
educational agencies in any State 
each year.... 

"...(d) Evaluation Models. - The 
(Secretary] shall provide to State 
educational agencies, models for 
evaluations of all programs conducted 
under this title. . .which shall 
include xiniform procedures and 
criteria to be utilized by local edu- 
cational agencies and State agencies, 
as well as by the State educational 
agency in the evaluation of such pro- 
grams. ... 

"...(e) Technical Assistance. - The 
[Secretary] shall provide such tech- 
nical and other assistance as may be 
necessary to State educational agen- 
cies to enable them to assist local 
educational agencies and State 
agencies in the development and 
application of a systematic evalua- 
tion of programs in accordance with 
the models developed by the [Secre- 
tary] . 



IMPACT AND ISSUES 

These new regulations clearly spoke 
to the need for providing systematic, 
comparable and valid data on the 
effectiveness of Title I progreons. 

In 197A, the U.S. Office of Education 
awarded a contract to initiate the 
development of an evaluation and 
reporting system that could be used 
by various consiu&ier's of evaluation 
data from Title I programs including 
local educational agencies, state 
agencies and the U.S. Office of Edu- 
cation. RMC Research Corporation, 
the contractor for the developaient of 
the models, was charged to develop a 
set of models, reporting formats and 
ether ancillary supporting forms, 
documents, materials and training 
strategies. 

Field work was conducted (1) to 
determine the current "state of the 
art" in terms of evaluation capa- 
bility at state and local levels and 
(2) to determine how the models would 
work in these settings. 

The sximmer of 1976 witnessed preli- 
minary work on the development of the 
Title I Evaluation and Reporting 
System (TI*.RS). At this point in 
time TIERS was not yet nationally 
implemented nor nationally mandated 
as the reporting system for Title I. 
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TABLE II (continued) 

Evaluation from 1974 - 1978 

"...(f) Specification of Objective 
Criteria. - The models developed by 
the [Secretary] shall specify the 
objective criteria wh.^h shall be 
utilized in the evaluation of all 
programs and shall outline techni- 
ques., .and methodology... for pro- 
ducing data v;hich are comparable on a 
S:Uit<)wide and nationwide basis. 

"•••(g) Report to Congress. - The 
[Secretary] shall make a [biennial] 
report to the respective committees 
of the Congress having jurisdiction 
over programs authorized by this 
title. . .concerning the results of 
evaluations of programs and projects 
required under this section, %rtiich 
shall be... based on the maximum 
extent possible on objective 
measures 

"...(i) Maximum Expenditures. - The 
[Secretary] is authorized to expend 
such sums as may be necessary to 
carry out the provisions of this sec- 
tion, but not to exceed one-half of 1 
per centum of the amount appropriated 
for such programs... In carrying out 
the provisions of this -section, the 
[Secretary] shall place priority on 
assisting States, local educational 
agencies, and State agencies to con- 
duct evaluations and shall, only as 
funds are available after fulfilling 
that purpose, seek to conduct 
national evaluations of the pro- 
gram." 
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TABLE III 



Evaliiaticn from 1978 - 1981 



EVALUATION REQUIREMENTS 

TITLE I EVALUATION AND REPORTING 
SYSTEM (TIERS) 

The objective of the Title I Evalua- 
tion and Reporting System (TIERS) was 
to provide meaningful, comparable 
information about Title I projects at 
the 



• 



school building level, 
school district level, 
state level, and 
federal level. 



Data were collected and/or a^^gregated 
on six topics: (1) participation, 

(2) parent advisory councils (PACs), 

(3) personnel, (4) training, 

(5) costs, and (6) impact. Impact 
data was gathered only from projects 
in the basic skill areas using one of 
the three evaluation models: 

Model A - The norm-referenced model 
Model B - The control group model 
Model C - The spe::ial regression Model 



IMPACT AND ISSUES 

The implementation of the models as 
was mentioned above provoked debate 
from the technical to the program- 
matic in all sectors. 

Operationally, even though 3 imple- 
mentation models for TIERS were 
developed. Model A was the only model 
in use. Use of the other recommended 
models, or other approved models was 
ext. jly rare accounting for only 1 
percent of the project evaluations. 

With respeci. to model A, cotv^ems 
were raised about the cumulative 
effects of statistical artifacts, 
unrepresentative norms, conversion 
errors, student selection, test 
administration procedures, attrition, 
and practice effects on the validity 
of estimates derived from TIERS. 

Desnite the technical concerns with 
the models and resistance in some 
cases to the implementation of the 
models, this forced intrusion of the 
federal government into the mandating 
of evaluation models at the national 
level forced state education agencies 
and local education agencies to take 
stock of their evaluation proce- 
dures. For some states, with strong 
evaluation units, TIERS posed no 
burden but for unsophisticated states 
relative to assessment and evaluation 
technology, the TIERS moved them from 
the darkness into the light. 
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TABLE IV 



Evaluation from 1981 to Present 



EVALUATION REQUIREMENTS EDUCATION 
CONSOLIDATION AND IMPROVEMENT ACT OF 
(1981) 

(d) Records and information 

"Each state educational ag^sncy shall 
keep such records and provide such 
information to the secretary as may 
be required for fiscal audit and pro- 
gram evaluation (consistent with the 
responsibilities of the secretary 
under this chapter)." (Section 
555(d) ECIA, Chapter I) 

(b) Application Assurance s 

"The application described in subsec- 
tion (a) shall be approved if it pro- 
vides assurances satisfactory to the 
State educational agency that tae 
local educational agency will keep 
such records and provide such infor- 
mation to the state education agency 
as may be required for fiscal audit 
and program evaluation (consistent 
with the responsibilities of the 
state agency under this chapter), and 
that the progran*" -.4id projects 
described." (oection 556(b), ECIA, 
Chapter I) 

Sustained Gains 

(A)".. .Will be evaluated in terms of 
their ef fectiventjss in achieving the 
goals set for them and that such 
evaluations shall include objective 
measurements of educational achieve- 
ment in basic skills and a determina- 
tion of whether improved cerformance 
is sustained over a period of more 
than one year." (Section 556(b)(A), 
ECIA, Chapter I) 

16801 



IMPACT AND ISSUES 

With the impact of ECIA, Chapter I 
evaluation requirements, which were 
minimal at best, state directors of 
Chapter I programs officially adopted 
a stand of continuing ^he implementa- 
tion of TIERS even though it was no 
longer mandated. The effort was 
laudable for a system now voluntary. 
As one can imagine, a system that no 
longer has binding controls begins to 
slacken. Informally, the consensixs 
of state directors, evaluators and 
the federal, state and local levels 
and practitioners, is that strict 
evaluation requirements of TIERS are 
being loosened due to the voluntary 
nature of the system. 

• Strict cut-offs are being 
violated. 

• Selection and pre-test 
rules are abrogated. 

• Quality control parameters 
have been slackened » 

• Data edits have been 
reduced . 

• Raw data are being crunched. 

• Sampling errors are 
occurring. 

• Negative gains are checked. 

• Positive gains are not- 
checked . 

Candidly, one questions the ability 
to generalize from the data at all, 
at a national level, given the 
problems cited above. 
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